Instrument the Rust backend with Prometheus metrics and create Grafan… - #76

Closes #68

I have successfully instrumented the Rust backend with comprehensive Prometheus metrics and created Grafana dashboards. Here's what was implemented:

Changes Made

1. Dependencies Added (websocket-server/Cargo.toml)

• metrics = "0.21" - Core metrics library
• metrics-exporter-prometheus = "0.12" - Prometheus exporter
• axum = "0.7" - HTTP server for protected endpoint
• tower = "0.4" and tower-http = "0.5" - Middleware support

2. Metrics Module (websocket-server/src/metrics.rs)

Comprehensive instrumentation covering:

• RPC metrics: Error rates, latencies (p50/p95/p99), timeouts by chain/operation
• Cache metrics: Hit ratios, sizes, evictions, hit/miss rates
• Inference metrics: Model computation durations, throughput
• WebSocket metrics: Active connections, message rates, subscriptions
• Actor metrics: Block processing times, queue depths, backpressure events
• Database metrics: Query durations, connection pool stats
• Alert metrics: Generation rates, broadcast durations

3. Protected Metrics Server (websocket-server/src/metrics_server.rs)

• HTTP server on port 9090 with /metrics endpoint
• Optional Bearer token authentication via METRICS_API_KEY environment variable
• Full test coverage for authentication scenarios

4. Integration

• Integrated metrics into chain_actor.rs for block processing and queue monitoring
• Integrated metrics into subscription.rs for WebSocket connection tracking
• Updated main.rs to initialize metrics exporter and start the protected server

5. Grafana Dashboards (deploy/grafana/)

Created 4 pre-configured dashboard templates:

• Overview Dashboard: System health overview with key metrics
• RPC Performance Dashboard: Detailed RPC latency and error analysis
• Cache Performance Dashboard: Cache efficiency and eviction monitoring
• WebSocket & Inference Dashboard: Connection stats and model inference metrics

6. Documentation (deploy/grafana/README.md)

Comprehensive guide covering:

• Dashboard descriptions and installation instructions
• Prometheus configuration examples
• Complete metrics catalog
• Environment variable configuration
• Alerting recommendations

Usage

Start the server:

cd websocket-server
cargo run

Access metrics:

• Unprotected: http://localhost:9090/metrics
• Protected (with API key): curl -H "Authorization: Bearer YOUR_KEY" http://localhost:9090/metrics

Configure Prometheus:
Add the WebSocket server as a scrape target pointing to localhost:9090/metrics

Import dashboards:
Upload the JSON files from deploy/grafana/ to your Grafana instance.